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ABSTRACT 

The impact of an evaluation design on the effective 
iaplenentation of a training program for women in higher eiucation 
leadership and management was assessed. The program was designed by 


the University of Pennsylvania's Higher Education Resource Services. 
The training program, entitled "The Next Move," was introduced at 


three pilot sites: a large, public urban institution, part of a 
multi-campus systema; a small, private university in a suburban 
setting; and a senior college of an urban public university systea. 
To increase chances for career mobility for women and to assist in 
the development of a positive self-image, the program utilized 
leadership and management training to vrovide women with an 


institutional perspective, strengthen and encourage local leadership, 


and foster the growth of networks. The program consisted 2f a 


one-half day case study workshop, a one-day leadership and management 
clinic, and°brown bag seainars. To insure implementation o£ the aodel 


on a nationwide basis, it was necessary that the evaluative design 
consider the special probleas of a model focusing on institutional 
change, while carefully appraising the potential adaptability or 
transportability of the program. The evaluation design selected was 
dlluminative evaluation, which encompassed both process and product 
components and the collection of quantitative and qualitative data. 
Because the program was designed in three stages, formativa 
evaluation was easily built into the evaluative model. Quantitative 
data were obtained through a questionnaire, administered after the 
“case study workshop and qualitative data were obtained through 
observations and interviews. Results of the evaluation are 
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Introduction 


g 


ria aie ats ask aaa erent oar eas 
the successful introduction of an innovative program whose long 
range impact can dramatically affect the institution. Such achiev~ 
ments of this kind are xare,due to the inherent problems of program 
design, as well as the difficulties of adequate, CELE oval 
uation. Clearly, when the program is sophisticated and change-ori- 
eetud. it ast ba petted by ani sotally comptes erative teins 
in order to accurately assess the program's’ impact. 

0 

This paper addresses itself to questions surrounding the de- 
sign and implementation cf an innovative program, and it examines 
the role of illuminative evaluation in assessing the success of 
the model. The program, The Next Move, is change-oriented and 
Ciimaw en ieee oe nen 1d tdi Base, 


. The structure of post-secondary education, in many respects, 

mirrors other institutions in our culture. ite oupantd wedel of aie 
aciky tints cet we ital ices, spatial by ae eae ee 

_ dn lower levels of authority and pover. Van Alstyne and Withers ns 
(1977) determined that only 16 per cent of all administrative po- 

gikicne 4 niche education ware Coeeled by Wouee: despite the 

‘fact that the majority of American post-secondary students are now 

female (Cowan, 1980). 
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- In previous projects, focussing on women in higher education, 
the program developer, of this study,had concluded that women's ca- 


‘ reers were hindered as much by their own negative self-perceptions 


as by institutional barriers. Women frequently believed that their 
professional skills were inadequate,even in the face of contrary e- 
idence. . 


The-Next Nove was designed to address the slow pace of career 
nobility for women in post-secondary institutions and the importance 
of developing a positive self-concept. ‘Tb increase chances for mo- 
pilibp and Abalist dn the develocuent of a peattive seif-image, the 
program utilized leadership and managenent training as a vehicle to: 
provide women with an institutional perspective:, strengthen and en- 


* courage local leadership: and foster.the growth of networks. 


The progcam developer and the external evaluator of The Next Mover 
in introducing the model on three pilot sites, felt that iliuminative 
evaluation would reveal the complexities of the model and highlight 
the strengths and weaknesses associated with its implementation. 
Tlluminative evaluation, as defined by Parlett and Hamilten (1978), 
“takes account of the wider contexts in which educational programs — 
functicn. Its primary concern is with description and interpretation _ 
rather than‘ neasurerent and prediction" (p. 10). Illuminative eval- 


, uation was expected to test the program's adaptability or transport- 


‘ . 
y 
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ability from site to site. s 


; Th this paper, ‘two major questions are considered: a 
(Q) Can a model training program for wonen adninistrators a 
‘be sufficiently transportable to meet the varied needs 
| of individual participants and the requirenents of par ~ 
ticular institutions? , : 
(2) Can the assessment appreach, known as "illuminative" eval- 
suit av artic “Kes eeaice~ Gb chun vicina wd ahaa tear 
4 portability of the original ‘model? ; 


Description of the Pilot Program = 
HERS (Higher Education Resource ores “Mid-Atlantic, an 


office housed at the University of Fenmlvenia, designed and im 
plemented The Next Move. The program consists of three parts: 
Part I is a ane and one-half day Case Study Workshop focussing on 
issues facing post-secondary women administrators; Part II is a 
one day Leadership and Menagenent Clinic designed to expand upon 
issuss identified in Part I; Part III is composed of five "Brom 
Bag" Seminars, held at tvo-week intervals, addressing topics se- 
lected by the participants. The workshop and clinic are conducted 
by facilitators and staff external to the institution, assisted by 
_ cofacilitators from the campus. A local coordinator selects 
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the co~facilitators, arranges for campus facilities, and invites 
woren participants. ‘these are drawn from four different grows: 
support staff, entry-lavel adunistration, midclevel administra- 
tion and facylty. 


. be 
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The strength of The Next Move, as a professional educaticn 
'. program, is derived from two.key design innovations: the inclus- 
ion of diverse waren fran all areas of the institution, and the 
tie ak tia oececeeitd GiLlete ites ie Gist up ata oes 
ticipants themselves. ‘The Next Move involves women in an extended 
- learning program emphasizing career development and an institution- 
al perspective. The participants, who traditimally share the same 
SEPIA Ey POE LEE Ia Riee ers encaura aM Se Aemnlce Menor md 
a sense of interdependence, 
the Nxt Yove was introduced to the Mid-Atlmtic region at three 
pilot campuses. These campuses were selected to challenge the model's 
desidn; thus, the institutions were intentionally diverse. Site A was 
a large public institution, part of a mlti-campus system, in an urban 
setting. Site B; a small, private miversity in a quiet susuxben sett- 
_ ing, was the only one of the three institutions currently experiencing 
expansion. Site C was the senior college of an urban public university 
system beset by declining enrolinent and drastic budget reductions. In 
Mey and dine, 1990, one-hundred and five wonen, at the three pilot in- 


stitutions, participated in Parts I and II of the model. 
; /? 
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“the potential adaptability or ee ee of the program. 
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In September, 1980, the Planning Cotmittee, at each of the sites, 
organized the third rhase of the program, ‘The Brown Bag Seminars, 
focussing on topics selected by the participants themselves. 

* fhe pilot study for ‘The Next Move has now been completed, and 
the program is available for nationwide introduction. 


Rationale ‘for. the Evaluation of the Pilot Program 

For implanting the model on a nation basis, it was essen- 
tidl that the. evaluative design consider the special problems of a — 
ses Weimetac wn Anaeheeii Gage while carefully appraising 


With respect to implementation of a model for change, such as 
ths iNext Move, current Literature stressed the need to focus on the 
process of planned innovation (Berman and McLaughlin, 1975). To’ 
facilitate implementation; The Next Move's design provided a period 
of time to elapse between each of its three stages; additionally ,it 
proceeded from an externally~administered to an intemally-directed 
program. The program also acknowledged the need for cn~site leader- 
ship by utilizing personnel from the host institution (Pincus and J 
Williams, 1979). Indeed, with the development of the program, the 
host campus’ coordinator and co-facilitatar ware requested to be- 
come more involved with The Next Move and they assumed responsibility 
for the program's continuation. 
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To effectively disseminate this program, the model emphasized 
the concept of "trangportability".: Used in this paper, "transport- 
ability" is close to the notic of “exportability" (Klein, 1974) or. 
the nore currently employed tem of "adaptability", i.e., "the ex- 
tent 5 which critical elements of an innovation.can be implemented 


at other sites" (Graeber, 1980, p. 14).° Fullan’and Pomfret (1977) 

noted two attributes related to trankpartability: "explicitness or 

plans for explicitness" and “complexity or degree of difficulty of 

change required by an innovation" (p. 368). ‘the evaluative design, 
to determine “transportability" of the model, would have to include | 
a study of the explicitness of the program and any necessary modifi- 
cation, as well as the degree and complexity of change as determined 
by the program's participants. 3 


Tlluminative Evaluation 
To judge the impact of The Next Move as a model for charge and 


to ascertain its transportability, the choice of an evaluative frame- 
‘work was a crucial decision. Tt was decided to use illuminative eval- 


uation. This mode of evaluation would best provide answers to the cam. 
ch i a a oe In 
addition, this fom of evaluatinn has been specifically recomended 
for the assessment of innovatory programs. : 


As defined by Parlett and Hamilton (1978), "illuminative evaluation 


. dia rik a whined Sew thodcinctendl packs) ub a General ebateay. It 


* glial ta ha oth acipkabsla: abl Mclaceda The choice of research 
‘tactics follows not from research doctrine, but fron decisions 


fines the methods used, not vice versa" (p. 13). 


in each case as to the best available techniques: the problem de- 
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. the components of the program envisioned by the designer, In fact, 


it seemed especially appropriate for two reasons: (1) since the _. 
sites were intended as pilots, the program developer wished to . 
have the widest range of response available to strengthen the model; 


(2) as a training program, The Next Move required that participents 


accept the program fully and agree to assume responsibility for the 
final phase of the model. ‘Accordingly, recognition of participants’ 


. diverse reactions to the model was necessary to assure the program's 


product components and conhining the collection of quantitative and 
qualitative data, ves dewiloped. Because The Next Move was designed 
in three stages, formtive eveluntion was easily built into the eval- 
uative model. "In-house" reports were provided by the evaluator to 


give continual Feedback to the program developer for: «, , . 


a) the next stage of the design, on a particular campus, so 
| that it could Better meet the needs of the participants; 
b) the refinenent of a particular stage which would then be 
implemented. on moter ste. : 


a 
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Quantitative data was elicited through a questionnaire, ad- 


| ministered after the vase Study Workshop (Part I)- at all three 


sites. “At the conclusion of the Brow Bag Seminars, further in- 
formation was’ collected throygh a final, questicnhsire. The com- 
‘ined information provided a pre-and post-test to detemtine if 
program objectives and participent expectations were achieved. 
At the beginning of each cycle, quantitative information was al- 
so obtained through a needd assessment which was completed, tab- 
ulated and returned to the cogrdinators at the individual sites. 
This information was used by campus organizers to develop.the di- 
rection and objectives for future infaractions: | i 
Qualitative data was obtained throuh observations and inter~ 
views. As a participant cbeerver, the evaluator attended all first 


and second cycle sessions. Field notes were taken and were used to 


provide relevant information to the project developer. After each 
stage of the model, the evaluator held telephone interviews, ‘using 
a structured fomat, with selected facilitators and participants. 


"Ruta was categorized and’ presented to the project developer. Trian- 


gulation was alao utilized to provide qualitative information, After 
“the initial Case Study Workshop, it was found to be’a useful mechanism 


; for assessment. In this instance, the ‘comments of extemal facilitators 


and internal group leaders were compared and contrasted. 
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Results é . . ‘ . 

; to detamine the success of Part 1, the Casé Stu Yorkshop, 
“vbuich was at the core of the program, partidipmts yere asked to me a 
indicate their satiafaction on the questiomaize. staal ea 


participmts who wire either “very” 2 Gbuaie satistied with “ 
' the.workshop (the two highest indicators), increased from site to 
_ site Le. Site A~42.0%; Site B-45.5%; Site C-65.7%), due.in large © 
part do atlerations in tho mode 'as a rule of particiomts' i? 
tb e% 

Dissatisfaction, sich was 2Slayed to the program developer pak 
_ through foemative evaluation centered around three ares discussed | 
” below: segregation of the groups, “responsibilities of the co-facil- 
itatts, and the workshop‘presentaticns of the prograni developer. - 

(ij segregation of the Groupe - The program developer sep- 
arated participants in Part. I-accortiing te their occupations, Par- 
tlpants at Site A cbjcted tranuusly to the segregation of rope 
and charged that *elitisn" had occured. Iware of this difficulty, : a7 ’ 
yet reluctant to chinge the original design, the program developér _ 
emounced at the nent two sites thet participants would meet in mb : 
ed groups in Part II, Using the formative evaluation information; | : 
and indicating the resulting change in participent mix, followup = _ \ 
interviews showed that segregation of groups was rip longer an “Laswe. ", 

(2) Responsibility of the Co-Facilitators - Implicit in The Next 
, Mowe destin was the gradual energenon‘of leadirahip and direction from - ee 


. 


OS at. 


" talks #ould be more easily renenbered if a sinmary were provided. ‘ 
‘Responding to her own perception of the presentations and to par- . # 


- to pursue. Although the majority of participents.at all sites se- 
. lected Management and Leadership Skills as their first topic of 


f ° 
the host institution, and it was hoped that the co-facilitators 
would evolve into key: leaders. However, after Part I, it was . 


_, “evident, that confacilitators on Site A felt uiprepared for the 
® yarkshop and had! begun to, worry about, their role in Part IT. At 
Site B, there was a further exmessiin of this andety. ny deal, 
with the crucial ‘issues of mile definition, the program developer 
described the co-facilitators' responsibilities at each stage of 
the model, and presented the outline prior to Part I. Followup “ 
interviews confimed' that! co-facilitators' concems vere allevi- 
ated onos their role in the model was mde-explicit. 
* (@) Workshop Presentations of the Program Developer‘ - The \ 


icici ivnlopek delivered tie Societe possaakishw tn ea’; 
Comments were made, both arally and on the questionnaire, that the ' 


_ticipants' reactions, the program developer produced an agenda, out-— 
“ning the talks and proving related tofies tor gra Ascusin. 

The questionnaire, administered after Part I, asked partici- ; 
pnts tis indicate professional develorment, areas vhich they wished a 


conc, the remaining develoment areas varied considerably fron 4 
site to site. or ‘ o , ’ * 5 
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ation of site specific topics of interest. 


“ Acoordingly, Part II of the model reflected the collective 


interests of the participants in a given institution. For ex- 


ample, at Site A, career plarining and human behavior skills had 
a high priority and were a major focus of discussion Site B, 
job desicn, supervision, performance assesement and conmnication 
skills received attuntion; on Site C, commmications and human be~ 


; havior skills were stressed. Thus, transportability was demn- 
. ¥ 


strated. ‘The model was sufficiently flexible to permit consider- 


the Next Hove's participants ware responsible for selecting — 
discussion topics for the Brown Bag Seminars (Part III). the mdel 
mixrored the previous activities and level of consciousness on an * oy 
individual campus and was therefore cc be adaptive. In 


_ this case, on one traditional campus, pa gaScindt nthe 


need to post available positions on a bulletin board, while on an- 
other more non-traditional site, participants indicated adesireto =~ 
organize a Women's Action Alliance. 


The adaptability of the model was also clearly demonstrated in 
the manner in which each campus changed the final stage of the pro- 
gram. The coordinator and her committee, at Site B, implenented 


| Part IIT as designed. The brown bag luncheons were:held two weeks 


apart and attendance was restricted to the original participants. 


o 
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For the luncheons, the theme of cammmications was selected and 
a seminar series relating to this topic was devised. Such titles 
as "Selling Ideas or Making the Boss Listen” and "Bring Your Boss 
to Launch" indicated clearly the thrust of these sessions. 


At Site A, there was a reduction ir che number of luncheons 
from five to three and colleagues were invited who were outside 
of the original program. n this campus, the brow bag sessions 
emphasized professional development. In this case, support systems 
for women; private sector jobs; and financial planning were the 
topics of the three luncheons. i 


" Qa Campus C, the Brown Bag Seminars were used as a spring- 
board for the formation of a new campus-wide organization of 
women. For the seminars, no central focus was noticeable. In- — 
stead, the monthly meetings ranged from a panel, discussion with 
four women deans, to a Christmas party, to as unusual a pre- 
sentation as "The Medieval Woman and Her Work." Attendance at 
each session was very good and the meetings were of interest to 
sacra, icon abl parte oF tha dintdernlty: Tt was noted that some 
of the original participants, in the secretarial group, as a 
irect outgrowth of the program, developed an informal suppart 
ae | 
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Discussion of the Results | 
Tha iat Teva Le a aaeceonal ly coiniee pea fee chaos. 

Variables abound. ‘hus, the sites are very different; women 

are at diverse stages of their professional development and at 

varied levels of consciousness; opportunities for individual 

growth on each campus differ. Despite the variables, the model 

this Fae eppanen 60 he tenieportabla; It can be adjusted both 

to the needs of participents end to the requirements of the dif- 

NV Evidence in support of the model's adaptability can be seen : 

in the willingness of the program developer to write more explicit 

material s‘Part 1) in the modification of Part II to sult the 

needs of the participants; in the variations which enabled par 

ticipants to tailor the luncheons in Part III to reflect their 

own interests. 


Illuminative Evaluation and Transportability 
The selection of illuminative evaluation to assess the im- 


pact of The Next Move proved to be of critical significance. Had 
only traditional evaluative methods been enployed, it is doubt- 

ful’ that the model could have been refined as it moved from site 
to site, or that the unintended ‘consequences of the program’s 


ee ee ee ee 
nized that for each intended action, there are some unintend- 
ed consequences (Magee, 1973). Tn a process model for change, 
wich ee Tha Nese Move niiriterin’ ie eneepoctod werpertinione 
were inevitable. Since participants were able to shape sore 
di tie peerranta pacts: the design was left deliberately open- 
ended. ne ee Ee ee ee ee 
eclectic, provided the external assessor with necessary method- 
ologies to properly describe the most obvious results of the 
program, as well as the unanticipated consequences. 


Although it was expected that a program, focussing on 
career development, would alter mobility and job enhancement 
for its participants, the program developer did not anticipate 
the speed at’ which changes would occur. Within ten months of 
the program's introduction, 20% of the one-hundred and five 
original waren reported positive career changes (i.e. new and 
expanded responsibilities, new titles, different posts, non~ 
scheduled salary increases). 


ae 
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It is difficult to establish a casual relaticnstiip be- 
tween participation in The Next Move and career changes; how- 
siver;: Spiced: cmt Sica the diecirert alts) wits uw the 
following, would strongly suggest. a causative relationship. 
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"It (The Next Move) helped me make up my mind to make a 
change in position of twenty-five years plus!" 

“ZI moved from Personnel to Law School. I would not have 
bid on the job last Spring." 

i Geer ice Ap Rig iney @ eacnieeh tem Tectnnel eleeins 
and up-grading of my current position." j 
"I have prioritized my responsibilities and resolved to 
take on more administrative work - to change direction and 
look at academic administration." 


It is interesting to note that followup questionnaires 
‘qnly partially elicited information on career mobility. It was 
telephone interviewing, one approach available in illuminative 
evaluation, which enabled the evaluator to learn of career 
Another interesting finding concemed the concept of net- 
working. Although the program was designed to foster the devel- 
opment of networks among participants, the extent to which waren 
at Site C would assist each other was unanticipated. this . 
campus, two support groups, with different but related functions, 


ees. Both groups have continued to meet often and offer advice 
and assistance to participants. Once again, questionnaire data 
did not demonstrate the formation of these two groups. Inter- 


views of the on-site coordinator and of past participants 
revealed this finding. 


In summary, this paper attempted to establish the relation- 
ship between effective implementation of an innovative program, 
The Next Move, and its evaluative design. ‘The Next Move is a 
process model for change and as such required an assessment de- 
sign in which product evaluation alone would not suffice. In 
addition, the evaluative design needed to be sufficiently flex- 
ible to cope with umpredictable outcanes. With this in mind, 
illuminative evaluation was "sed to provide for the program 
developer the informition necassary:to modify the model from 
stain to abage cod Pom wite toults, ti thie sey, the wcdel 
iia able ea bann temecortala by peak acre e1esel ha Sad 


needs of individual participants, as well as the requirement of . < 
' particular institutions. Tluninative evaluation also revealed . 
unintended consequences, which helped rection the program's 
positive impact on career mobility and on the development of 
support aroups. 
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